When to say “Don’t Know”: Confidence in Automatically Generated Hypotheses without the Assumption of an Underlying Distribution

We have a set S ⊂ {0,1}n, together with, for each x ∈ S, the result of some unknown function F : {0,1}n → {0,1} applied to x, and a method for generating a hypothesis h ∈ H about F given S. We present theoretical and experimental results on four possible methods (similarity, convexification, prevalence and Hamming distance) for determing, given two elements y,z ∈ {0,1}n\S, whether we should be more confident that h(y) = F(y), or that h(z) = F(z), or indeed that we should attach the same degree of confidence to both statements. We consider whether it is possible to have an absolute measure of confidence in the statement that h(a) = F(a) for any given a ∈ {0,1}n\S. We introduce a modification of a standard learning algorithm for Boolean functions, which naturally partitions new examples into three categories: 1,0 and don’t know.